Moju Kapu モジュカプ

How mirai and crew Are Powering the Next Generation of Parallel Computing in R

Charlie Gao and Will Landau

Hibiki AI Limited, Eli Lilly and Company

2024-07-09

mirai

未来

みらい       / mI ˈ ra ˈ i: /

  1. future

library(mirai)

a <- 1
b <- 2

m <- mirai(
  {
    Sys.sleep(5)
    a + b
  },
  environment()
)

m
#> < mirai | $data >

m$data
#> 'unresolved' logi NA

m[]
#> [1] 3

m$data
#> [1] 3

But what’s so special about this? …

  1. Highly performant
  2. Simple and robust
  3. Designed for production

Integration with Base R

Request by R Core (Luke Tierney) at R Project Sprint 2023

  • mirai added as the first alternative communications backend for the base parallel package
library(parallel)

cl <- mirai::make_cluster(2)
cl
#> < miraiCluster | ID: `0` nodes: 2 active: TRUE >

parLapply(cl, 1:2, rnorm)
#> [[1]]
#> [1] -0.5697883
#> 
#> [[2]]
#> [1] 0.3798926 0.8945085

Integration with Shiny

library(shiny)
library(bslib)
library(mirai)

ui <- page_fluid(
  numericInput("n", "Sample size (n)", 100),
  numericInput("delay", "Seconds to take for plot", 5),
  input_task_button("btn", "Plot uniform distribution"),
  plotOutput("plot")
)

server <- function(input, output, session) {
  task <- ExtendedTask$new(
    function(...) mirai({Sys.sleep(y); runif(x)}, ...)
  ) |> bind_task_button("btn")
  observeEvent(input$btn, task$invoke(x = input$n, y = input$delay))
  output$plot <- renderPlot(hist(task$result()))
}

app <- shinyApp(ui = ui, server = server)
with(daemons(3), runApp(app))

Integration with Plumber

library(plumber)
library(promises)
library(mirai)

pr() |>
  pr_get(
    "/echo",
    function(req, res) {
      mirai(
        list(status = 200L, body = list(msg = msg)),
        msg = req[["HEADERS"]][["msg"]]
      ) %...>% (function(x) {
          res$status <- x$status
          res$body <- x$body
        })
    }
  ) |>
  pr_run(host = "127.0.0.1", port = 8985)

crew

Re-encapsulating mirai

Why crew?

  • Extends mirai to distributed computing environments.
  • Centralized R6 interface for tasks.
  • Worker auto-scaling to respond fluctuating task loads.

Moju Kapu in crew

.pull-left[

Encapsulation

  • Uses mirai developer interface: daemon(), nextget(), saisei() etc.
  • R6 class system for the controller interface.

]

.pull-right[

Modularity

  • Plugin system to launch parallel workers on different environments.
  • Docs guide users to write their own plugins.
  • Existing encapsulated plugins for SLURM, AWS Batch, etc.

]

R6 classes

Class About
Controller group                 Interface for many controllers.
Controller Central task interface.
Tasks List of mirai task objects.
Client mirai TCP connection hub.
Relay Coordinate mirai synchronization primitives.
Launcher Launches workers. Subclasses for plugins.
Async Parallelize worker launches.

R6 object composition

mirai tasks, different interface

Different plugin, different controller

Write your own launcher plugin

Controller wrapper